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Partially Real and Partially Simulated Modular 
Interactive Environment 

Cross-reference to Related Applications 

[0001] This application is based upon and claims the benefit of the filing date of U.S. 
provisional application, serial number 60/406.281 , filed August 27, 2002, entitled 
"Immersive System of Digital Walls and Physical Props," the entire content of which is 
incorporated herein by reference. 

Background of Invention 
[0002] Field of Invention 

[0003] This application relates to virtual reality systems, including virtual reality 
systems that simulate a real environment. 

[0004] Description of Related Art 

[0005] It is often helpful to simulate a real environment to test one or more individuals 
or to train them to better cope with the real environment. 

[0006] Conducting the test or training in the real environment is often more realistic. 
However, it is also often impractical. The real environment might pose dangers to the 
individuals, such as in a hostage situation or the repair of a contaminated nuclear 
reactor. Failure of the individual to succeed with his effort in the real environment 
might also lead to catastrophic losses. The real environment might also not be readily 
accessible or available. 

[0007] Mock-ups have been made as part of an effort to meet this need. Sometimes, 
however, the mock-ups take too long to create, cost too much money, are difficult to 
reconfigure, and/or are hard to transport. Mock-ups also sometimes lack a sense of 
realism. 

[0008] The projection of images on screens has also been used. These 
presentations, however, also often lack realism. There is usually nothing real to touch 
or interact with, and the screen is often devoid of any surrounding context. 
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[0009] Head-mounted displays have also been used. Still, however, there is often 
nothing real to touch. These devices can also be heavy and seriously restrict 
movement. 

Brief Summary of Invention 

[0010] An interactive environment may be partially real and partially simulated. It may 
include a structure that is large enough to accommodate an individual both before and 
after the individual takes a plurality of displacement steps within the structure. At least 
one real, three-dimensional object may be positioned within the structure, the real 
object and the structure cooperating to form a seamless and integrated scene. At least 
one computer-controlled display may be fixedly positioned within the scene such that 
each image displayed by the display appears to the individual to be real, three- 
dimensional and an integral and seamless part of the scene. At least one sensor may 
be configured to sense interaction between the individual and the scene. A processing 
system may be in communication with the sensor and the display and configured to 
deliver a sequence of images to the display, the content of which are a function of the 
interaction between the individual and the scene, including a plurality of displacement 
steps taken by the individual. 

[001 1] The display may be configured and positioned so as to appear to the 
individual to be an integrated and seamless part of the scene that is something other 
than a display. The display may be part of the structure. 

[0012] The display may include a wall of the structure. One of the images may 
include an image of wall texture, and the processing system may be configured to 
deliver the image of the wall texture to the display. 

[0013] A real, operable door may be positioned in front of the wall of the display. 

[0014] A real window may be positioned in front of the wall of the display. The 
window may be operable such that it can be opened or closed. The window may 
include operable shutters. 

[001 5] The interactive environment may be configured to simulate a real environment 
having a similar structure, a similar real object and a scene beyond the structure. One 
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of the images may include an image of the scene beyond the structure. The interactive 
environment may include a real or virtual door or window. At least a portion of the 
display may be oriented within the opening of the door or window. The processing 
system may be configured to deliver the image of the scene beyond the similar 
structure to the display. 

[0016] One or more of the images may be selected by the processing system from a 
library of images stored on an image storage device. 

[0017] The interactive environment may be configured to simulate a real environment 
and one or more of the images may be captured from the real environment. The 
images that are captured from the real environment may be delivered by the 

processing system to the display in real time. 

[0018] The structure, display and real object may be configured in the form of 
modules that releasably connect to and disconnect from one another to facilitate the 
assembly, disassembly, shipment and re-assembly of the interactive environment in 
various different configurations. A portion of the structure, the display or the real object 
may include wheels for easy transport. 

[0019] The structure may be large enough to accommodate a plurality of individuals 
both before and after each individual takes a plurality of displacement steps. 

[0020] The structure, display and real object may cooperate to create the 
environment of a room. 

[0021] The structure, display and real object may cooperate to create the 
environment of a building having a plurality of rooms. 

[0022] The structure, display and real object may cooperate to create the 
environment of a land, sea or air vessel, 

[0023] The interactive environment may include a computer-controlled sensory 
generator, other than a display, configured to controllably generate matter or energy 
that is detectable by one or more of the human senses. The processing system may 
also be in communication with the generator and configured to control the generation 
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of such matter or energy. The processing system may be configured to control the 
generator as a function of the interaction between the individual and the scene. 

[0024] The generator may include sound-generating apparatus. 

[0025] The generator may include movement-generating apparatus. The movement 
generating apparatus may include floor movement generating apparatus. The 
movement generating apparatus may include air movement generating apparatus. 

[0026] The generator may include a light. 

[0027] The generator may include temperature-changing apparatus. 

[0028] One or more images may be stereoscopic images. The display may be 
configured to display the stereoscopic images. 

[0029] A distributed interactive environment may be partially real and partially 
simulated. 

[0030] The distributed interactive environment may include a first computer- 
controlled display fixedly positioned within a first scene such that each image on the 
first display appears to a first individual in the first scene to be real, three-dimensional 
and an integral and seamless part of the first scene. A first sensor may be configured 
to sense interaction between the first individual and the scene. 

[0031] The distributed interactive environment may include a second computer- 
controlled display fixedly positioned within a second scene such that each image on 
the second display appears to a second individual in the second scene to be real, 
three-dimensional and an integral and seamless part of the second scene. The second 
scene may be substantially the same as the first scene, but separated geographically 
from the first scene. A second sensor may be configured to sense interaction between 
the second individual and the second scene. 

[0032] The distributed interactive environment may include a processing system in 
communication with the first and second displays and the first and second sensors. 
The processing system may be configured to deliver a sequence of images to the first 
display, the content of which are a function of the interaction between the second 
individual and the second scene. The processing system may also be configured to 
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deliver a sequence of images to the second display, the content of which are a 
function of the interaction between the first individual and the first scene. 

[0033] A modular, interactive environment may include a set of modular walls that 
releasably connect to and disconnect from one another to facilitate the assembly, 
disassembly, shipment and re-assembly of the interactive environment in various 
different configurations. At least one of the walls may be a computer-controlled display 
configured such that images on the display appear to an individual within the 
environment created by the walls to be real, three-dimensional and an integral and 
seamless part of the environment. At least one sensor may be configured to sense 
interaction between an individual and the environment created by the modular walls. A 
processing system may be in communication with the sensor and the display and 
configured to deliver images to the display that vary based on the interaction between 
the individual and the environment. 

[0034] These, as well as other objects, features and benefits will now become clear 
from a review of the following detailed description of illustrative embodiments and the 
accompanying drawings. 

Brief Description of Drawings 

[0035] Fig. 1 is an interactive environment that is partially real and partially 
simulated. 

[0036] Fig. 2 is a lateral perspective view of the display wall shown in Fig. 1 with a 
real door in front of it. 

[0037] Fig. 3 is a lateral perspective view of the display wall shown in Fig. 1 with a 
real window in front of it. 

[0038] Fig. 4 is an interactive environment that is partially real and partially simulated 
of a different configuration. 

[0039] Fig. 5 is a distributed interactive environment that is partially real and partially 
simulated. 

[0040] Fig. 6(a)-(c) are configurations of other interactive environments that include 
vessels. 
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Detailed Description of Illustrative Embodiments 

[0041] Fig. 1 is an interactive environment tliat is partially real and partially 
simulated. 

[0042] As shown in FiG. 1 , a structure 101 may include real walls 103 and 105. 
These may be constructed from props used in movie sets, i.e., from material that looks 
and acts like a wall, but is lighter, possibly thinner, and less expensive. 

[0043] The structure 101 may also include display screen walls 107 and 109. Display 
screen walls 107 and 109 may be configured to look like walls, as illustrated in Fig. 1 . 

[0044] The walls 103, 105. 107 and 109 may be configured so as to create the 
appearance of a room. The room may be large enough to accommodate an individual 
111 both before and after the individual 111 takes a plurality of steps displacing 
himself horizontally, vertically or in some other direction. 

[0045] At least one real, three-dimensional object may be positioned within the 
structure 101, such as a desk 113 and a table 115. The real objects may cooperate 
with the structure 101 to form a seamless and integrated scene from the perspective of 
the individual 111. 

[0046] The interactive environment may include one or more computer-controlled 
displays. In Fig. 1, two such computer-controlled displays are illustrated. One is a 
projector 117 directed toward the display screen wall 109. The other is a projector 119 
directed toward the display screen wall 107. Each of these make up a rear-projection 
display. In each case, the display screen wall may consist of or include an opaque 
surface that radiates a quality image on the surface opposite of the surface that is 
illuminated by its associated projector. 

[0047] Each computer-controlled display may be fixedly positioned relative to the 
stmcture 101 such that the display does not move during movement of the indivudal 
111. Indeed, each computer-controlled display or, as shown in Fig. 1, a portion of the 
display may be part of the structure, as illustrated by the display screen walls 107 and 
1 09 forming walls of the structure 101 . 
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[0048] The computer-controlled display or displays may be configured and positioned 
so as to appear to the individual 111 as an integrated and seamless part of the scene 
that is created by the structure and real objects. This part of the scene may be 
something other than a display, such as the walls illustrated in Fig. 1. 

[0049] The scene created by the structure 101 and the real objects 113 and 115 may 
be augmented by other real objects to enhance the realism of the scene to the 
individual 111. These other real objects may cooperate with the scene so to form a 
seamless and integrated part of the scene, again from the perspective of the individual 
111. 

[0050] One such other real object may be a real, operable door 121 , a reference 
intended to also include the frame of the door. This may provide the individual 111 with 
a mechanism for entering the room that has been created and to enhance the realism 
of that room. 

[0051] A still further real object may be a real, operable door 123, a reference 
intended to also include the frame of the door. This door may be positioned 
immediately in front of the display screen wall 107, forming a seamless and integrated 
part of the scene from the perspective of the individual 111. Fig. 2 is a lateral 
perspective view of the display screen wall 107 shown in Fig. 1 with the real door 123 
in front of it. The real door 123 is operable and thus may actually be opened by the 
individual 111, it being shown in open positions in both Figs. 1 and 2. When the 
individual does open the door, the portion of the display screen wall 107 that lies 
behind the door may then becomes viewable to the individual 1 1 1 . If , as described in 
more detail below, an image of a scene 227 is projected on the portion of the display 
screen wall 107 that is behind the door 123 by the projector 119, the individual 111 will 
see that scene and perceive it to be an integrated and seamless part of the scene that 
is created by the structure 101 and the real objects 113, 115 and 123. Of course, the 
individual 111 may not actually be able to step through the real door because of the 
display screen wall 107 behind it. 

[0052] Returning to Fig. 1 , a still further real object may be real window 125 placed in 
front of a display screen wall, such as the display screen wall 109. Again, the real 
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window 125 may be constructed and positioned with respect to the structure 101 and 
the other real objects so as to be seamlessly integrated within the scene, again from 
the perspective of the individual 111. And again, the individual 111 may not actually be 
able to extend a hand through the window 125 because of the display screen wall 109 
behind it. 

[0053] Fig. 3 is a lateral perspective view of the display screen wall 109 with the real 
window 125 in front of it. As shown n Fig. 3, the real window 125 may include real 
shutters 327, each of which may include adjustable louvers 329 and pull knobs 331. 
Similar to the door 123, the individual 111 will see images projected by the projector 
117 on the portions of the display screen wall 109 that are directly behind the window 
125 if the individual 111 opens a shutter by pulling on its knob 131 or adjusting its 
louvers 129. 

[0054] Returning to Fig. 1 , the interactive environment may also include one or more 
sensors, such as the sensor 127. The sensors may sense interaction between the 
individual 111 and the scene by detecting and/or measuring that interaction. 

[0055] The interaction that the sensors detect or measure may precede the several 
displacement steps of the individual, may follow those steps, or may occur during the 
steps. Indeed, the interaction that is sensed may be the steps themselves. 

[0056] The sensors may sense movement of the individual. In this case, the sensor 
could be a camera and associated signal interpretive system, an ultrasonic sensor, a 
UV sensor, or any other type of movement sensing device. The individual could also 
be instrumented with the movement sensor, such as a GPS receiver and an 
associated transmitter for transmitting the extracted coordinate information. 

[0057] The sensors may sense contact of the individual with an object in the scene or 
proximity of the individual to that object. For example, the sensors might detect that a 
door or window has been opened or that a gun has been shot. In the gun shot 
example, the sensors might include a sensor on the portion of the display screen wall 
107 that is behind the door 123 that sensors the location at which a laser beam 
intersects that portion of the screen. The sensors might include a sensor attached to a 
laser gun (not shown) that is held by the individual 111 that senses the location of the 
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laser gun in three dimensional space. Using well-known triangulation techniques, the 
aim of the laser can then be computed from the information provided by these two 
sensors, followed by a determination of whether a particular target that might be a part 
of the image behind the door 123 has been hit. 

[0058] The sensors may be configured to sense sound that is generated by the 
individual, such as by his voice or foot movement. 

[0059] The sensors may be configured to sense the level of lighting in the room set 
by the individual 111 and hence the anticipated visibility of the individual 111 through a 
window or door opening. 

[0060] The sensors may be configured to sense any other type of interaction 
between the individual and the scene. 

[0061] Although a single sensor 127 is shown as being located in the corner of the 
structure 101 in Fig. 1 , this is just to illustrate one example. Both the number and 
location of the sensors may vary widely, as suggested by the discussion above. 

[0062] A processing system 131 may be included and configured to be in 
communication with the projectors 117 and 119 and the sensors, such as the sensor 
127. The processing system 131 may also be in communication with an image storage 
system 133. The processing system 131 may also be in communication with a source 
of real-time image information over a communication link 135. 

[0063] The processing system 131 may be configured to select and deliver 
appropriate images to the projectors 117 and 119 from the image storage system 133 
and/or the source of real-time imagery over the communication link 135. The 
processing system 131 may be configured to cause the imagery that is selected to be 
delivered to the projector 117 and/or 119 to be a function of the interaction between 
the individual and the scene, including the plurality of displacement steps taken by the 
individual, as detected and/or measured by the sensors, such as the sensor 127. In 
this configuration, the sequence of images that is delivered to the projector 117 and/or 
the projector 1 19 are considered to be "non-linear" because the sequence is not wholly 
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pre-determined, but rather a function of the interaction between the individual 111 and 
the scene, as detected by the sensors, such as the sensor 127. 

[0064] The configuration that has thus-far been described in connection with Fig. 1 
provides a broad variety of flexibility and associated effectiveness in the interactive 
environment that is simulated. 

[0065] In connection with a military training exercise, for example, all of the louvers 
329 in the shutters 327 in the window 125 may be in the closed position with the 
individual 111 standing by the real desk 113. The individual 111 may then walk over to 
the window 125 and open the louvers 329 on one of the shutters 327. 

[0066] This opening may be detected by a sensor (not shown in Fig. 1 ). Upon this 
detection, the processing system 131 may be programmed to select a video of a 
sniper moving in position to fire from the library of images on the image storage device 
133. It may also be programmed to deliver this image to the projector 117 such that 
this video is seen by the individual 111 while looking through the open louvers 329 in 
one of the shutters 327 in the window 125. 

[0067] The individual 111 may then raise a laser gun (not shown), point it toward the 
virtual sniper, and fire. This interaction between the individual 111 and the scene may 
be sensed by other sensors, including the direction that the gun was fired. The 
processing system 131 may then compute whether the direction of the firing was 
sufficiently accurate to immobilize the sniper. 

[0068] To do this, the processing system may need information about the location of 
the sniper. This information might be developed from the image supplied by the image 
storage system 133 or from independent data that is supplied along with this image or 
as part of it. 

[0069] ^ If the processing system 131 determines that the aiming was sufficiently 
accurate to immobilize the sniper, the processing system 131 may then a video of the 
sniper falling to the ground select from the image storage system 133 and deliver that 
image to the projector 1 17, thus communicating to the individual 111 that his firing was 
successful. 
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[0070] On the other hand, if the aim was not accurate, the processing system 131 
might select a different image from the image storage system 133, such as an image 
of the sniper firing back. Information from the sensors about the exact location of the 
individual 111 may then be delivered to the processing system 131 at the moment that 
the sniper fires, thus enabling the processing system 131 to compute whether the 
individual was successful in moving out of the way in sufficient time. The determination 
of this computation by the processing system 131 may be communicated to the 
individual 1 1 1 or to others as part of the training or assessment of the individual 111. 

[0071] As noted above, the display screen walls 107 and 109 may be large enough 
to appear to the individual 111 as an entire wall. In this configuration, the processing 
system 131 may be configured to deliver to the projectors 117 and 119 imagery of the 
desired interior texture of the walls 107 and 109, such as the appearance of bricks. 
This would cause the display screen walls 107 and 109 to appear to the individual 111 
as real, brick walls. 

[0072] Of course, the imagery that the individual 111 sees through the door 123 and 
the window 125 would most likely be different. To accomplish this, the image that is 
stored in the image storage system 133 may actually consist of two images, the image 
of the textured area of the walls outside of the door 123 and window 125 and the 
virtual scene image that needs to appear in the portals of the door 123 or window 125. 

[0073] Alternatively, an overlay system may be used to overlay the images that 
should be seen in the portals on top of the texture images that appear around these 
portals. The needed texture could be separately stored as an image, such as in the 
image storage system 133. A single sub-area of the needed texture might instead be 
stored and replicated by the processing system 131 in the needed areas. 

[0074] Of course, other imagery could be projected on the portions of the walls 
outside of the portals created by the door 123 and the window 125. For example, a 
video of a tank breaking through the wall might be projected in appropriate 
circumstances. 

[0075] A broad variety of images may be delivered to the displays. They may include 
still pictures, videos, or a mixture of still pictures and videos. The images may be 

LAS99 1304220-4.028080.0109 "1 1 - 



28080-109 



virtual images generated by a computer or real images. They may be stored in 
electronic format, such as in a digital or analog format. They may also be optically 
based, such as from film. 

[0076] The images may be such as to appear to be real, three-dimensional and 
seamlessly integrated into the scene when displayed from the perspective of the 
individual 111. 

[0077] By combining all these features, an individual can effectively walk or run in the 
simulated environment and interact with real and virtual components in the 
environment, just as though all of the components were real. This effectively creates a 
mixed reality world where the real and virtual are integrated and seamless co-exist. 

[0078] The scene that is created by these components, including the structure 101 , 
the display screen walls 107 and 109, the real objects 113, 115, 123 and 125, and the 
images that are delivered to the display screen walls 107 and 109 may cooperate to 
simulate a real environment, such as the real environment 145 shown in Fig. 1. In this 
situation, the structure 101 and the real objects may be selected and configured to 
substantially match the corresponding components in the real environment 145, as 
shown by the inclusion in the real environment 145 in Fig 1 of similar-looking 
corresponding components. 

[0079] Similarly, the images that are delivered to the display screen walls 107 and 
109 may mimic what the individual 111 would see if he were in the real environment 
and had been interacting with the real environment as the individual 111 is interacting 
with the simulated environment. 

[0080] To accomplish this, photographs and videos of the real environment may be 
taken and used in the construction of the simulated environment and in the creation of 
the library of images that are stored in the image storage system 133. 

[0081] It may also be possible to position cameras in the real environment, such as 
cameras 141 and 143 in the real environment 145. The camera 141 may be pointed in 
the very direction of the imagery that the projector 117 may be directed by the 
processing system 119 to project in the opening of the window 125. Similarly, the 
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camera 143 may be pointed in the very direction of the imagery that the projector 119 
may be directed by the processing system 131 to project on the opening in the door 
123. The signals from the cameras 141 and 143 may be delivered to the processing 
system 131 over the communication link 135, as also shown in Fig. 1. 

[0082] A computer-controlled sensory generator 151 may also be included in 
association with the structure 101. Although illustrated as a single device in the corner 
of the room, it is to be understood that the sensory generator 151 may, in fact, be 
several devices located at various locations. 

[0083] The sensory generator 151 may be configured to controllably generate matter 
or energy that is detectable by the human senses, such as by the human senses of 
the individual 111. The processing system 131 may be in communication with the 
sensory generator 151 so as to control the sensory generator to add additional realism 
to the simulated environment. The control of the sensory generator may be a function 
of the interaction between the individual 111 and the scene, as sensed by the sensors, 
such as the sensor 127. 

[0084] The sensory generator 151 may include sound-generating apparatus, such as 
immersive audio systems and echo-generating systems. These can replicate the 
sounds that the individual 111 would expect to hear from the images that are being 
projected on the display screen walls 107 and 109, such as gunfire, thunder, voices, 
the rumbling of a tank, and the whirling of a helicopter blade. 

[0085] The sensory generator 151 may also include movement-generating 
apparatus, such as devices that shake the walls or floors of the structure 101. again to 
reflect what might be expected to be felt from many of these projected images. 

[0086] The movement-generating apparatus may also move objects in the room, 
such as causing pictures on a wall to fall, glass to break, or even the floor to move 
violently to simulate an earthquake. 

[0087] The movement-generating apparatus may also cause air to move, such as to 
simulate wind. 
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[0088] The sensory generator 151 may also generate particulate material, for 
example, droplets simulating rain or a mist simulating fog. 

[0089] The sensory generator 151 may also include one or more lights, such as one 
or more strobes to assist in the simulation of lightning or an explosion. 

[0090] The sensory generator 151 may also include temperature changing apparatus 
so as to cause temperature changes, providing even further flexibility in the types of 
environments that may be simulated. 

[0091] The components that interact together to create the interactive environment 
may be designed in the form of modules. The same set of modules may be used to 
create a vast variety of different types of simulated environments. 

[0092] The use of projected wall textures may be particularly helpful by allowing the 
same display screen wall to be used with different projected images, so as to create 
the virtual appearance of a broad variety of different types of walls. 

[0093] The modules may also be fitted with releasable connections so that one 
module may be readily attached to and detached from another module. This may 
facilitate the ready assembly, disassembly, packaging, shipment and reassembly of a 
broad variety of simulated environments from the same set of modules. 

[0094] As is well known in the art, a broad variety of connection systems may be 
used to implement this releasable function. Such systems include latching 
mechanisms and deformable male projections and female receptacles that allow 
modular-like components to be easily snapped or slid together and just as easily 
snapped or slid apart. Traditional fasteners, such as nuts and bolts, may also be used. 

[0095] The size of each modular component may also be kept to a specified 
maximum. If this is exceeded by the needed size of a large wall, the large wall may be 
made up by releasably connecting a set of smaller wall segments. 

[0096] Heavy pieces may also have wheels placed along their bottom edge, such as 
wheels 231 shown in Fig. 2 or wheels 341 shown in Fig. 3, to make it easier to 
relocate them. 
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[0097] Fig. 4 illustrates an interactive environment that is partially real and partially 
simulated of a different configuration. As shown in Fig. 4, the interactive environment 
includes rooms 401, 403, 405 and 407, each with associated windows 409, 411, and 
413 and 421 and doors 415, 417, 419 and 421. Display screen wall 423, 425, 427, 429 
and 430 may also be included and, in conjunction with projectors 431 , 433, 435, 437 
and 439, may make displays of the type discussed above. 

[0098] The specific details of this embodiment may follow one or more of the details 
discussed above in connection with Figs. 1-3. When constructed out of the type of 
modular components discussed above, the simulated environment in Fig. 4 can be 
easily and quickly constructed out of these very same components and with no added 
expense. This illustrates the great flexibility and power of the modular system 

[0099] Fig. 5 illustrates a distributed, interactive environment that is partially real and 
partially simulated. As shown in Fig. 5, a first structure 501 may include a display 
screen wall 503, a real window 505 and a desk 507. An individual 509 may be within 
the structure 501. The interaction between the individual and the scene created by the 
structure 501 and its associated real objects may be sensed by a sensor 511. A 
projector 513 may be directed to the display screen wall 503 creating in combination 
with the display screen wall a computer-controlled display. 

[0100] A second and complementary environment may include a complementary set 
of components, including a second structure 525, desk 527, real window 529, projector 
531, display screen wall 533 within the window 529 and sensor 535 configured to 
sense the interaction between a second individual 537 and the environment created by 
the second structure 525 and its associated components. 

[0101] A processing system 515 may be in communication with the projectors 513 
and 531 and the sensors 511 and 535. 

[0102] The configuration shown in Fig. 5 can be useful in training or testing a group 
of individuals, without requiring the group to be at the same location. 

[0103] All of the individuals may interact with the same mixed reality environment. In 
this situation, all of the corresponding components between the two environments may 
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be the same or substantially the same, to the extent possible. Although only two 
distributed environments and two individuals have been shown in Fig. 5. it is to be 
understood that a different number of either or both could also be used. 

[0104] All of the components shown in Fig. 5 may be configured to operate as 
described above in connection with the components shown in Figs. 1-4. To 
accomplish interaction, however, the processing system 515 may be configured to 
cause images that are sent to the projector 513 to be a function of the interaction 
between the second individual 537 and the scene he is associate with, as detected 
and/or measured by the sensors 535. Similarly, the sequence of images that the 
processing system 515 delivers to the projector 531 may be a function of the 
interaction between the first individual 509 and the scene with which he is associated, 
as detected by the sensors 511. The processing system 515 may also take into 
consideration the signals generated by the sensor that is in the same scene as the 
projector in selecting the image to be displayed in that scene. 

[0105] One application of this distributed environment is the sniper example 
discussed above. Instead of the sniper being a wholly virtual image, however, that 
sniper may actually be the real individual 537 shown in Fig. 5. As can readily be 
imagined, the processing system 515 may cause an image of the first individual 509 to 
be projected on the screen display 533, creating a virtual target for the second 
individual 537. Conversely, the processing system may cause an image of the second 
individual 537 to be cast upon the display screen wall within the portal of the window 
505, creating a target for the first individual 509. As each of the individuals react to 
their respective targets, the interaction between the individuals and their scene can be 
detected and/or measured by the sensors 511 and 535, respectively, causing the 
processing system 515 to appropriately adjust the next set of images that are 
delivered to the display being viewed by each individual. Because of the seamless 
integration of real and virtual components in each environment, the mixed reality 
systems may provide the illusion to each individual of a completely real environment. 

[0106] The images that are delivered to the projectors 513 and 531 may be 
synchronized in some manner to one another, such as the delivery of identical images 
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being delivered to both projectors. The selection and/or the tinning of the images may 
also be a function of a stimulus other than or in addition to the stimulus detected by the 
sensors 511 and 535. For example, the stimulus could be a signal from a trainer that is 
monitoring an interactive environment that is configured for training that initiates the 
simulation of an explosion. 

[0107] Although one or more rooms have thus-far been illustrated as the structure of 
the environment, it is to be understood that a broad variety of different structures can 
be created and operated using the same principles. These include buildings with 
multiple floors and multiple rooms, geographic areas that contain several buildings, 
alleyways, streets and associated structures, and vessels, such as the ship, plane or 
freight truck shown in Figs. 6(a), (b) and (c). respectively. 

[0108] Although only a single individual has thus-far been shown in each structure, it 
is to be understood that each structure could be large enough to accommodate a 
plurality of individuals, both before and after each takes several displacement steps, 
either horizontally, vertically or in some other direction. The structures may even be 
large enough to allow the individuals to walk many steps or to even run. 

[0109] Although certain real objects have thus-far been illustrated to enhance the 
realism of the scene, it is to be understood that other real objects could also be used, 
such as pictures, floors, fans, lighting, carpeting, bookshelves, wallpaper and ceilings. 
All of the real objects may also be part of the modular system discussed above, 
making them easy to assemble, disassemble, transport and reassemble. 

[0110] Although only rear-projection displays have thus-far been illustrated, it is to be 
understood that any type of display can be used, including front projection displays 
and flat displays, such as large plasma displays and electronic ink devices. 

[0111] The image that is delivered to a display may also be a stereoscopic image 
suitable for generating a three-dimensional effect. The display that receives this image 
may also be one that is suitable for displaying the stereoscopic image so as to create 
the three-dimensional effect. The image that the individual views within the scene may 
then appear to change in perspective as the individual changes her position relative to 
the display, thus adding to the realism. The individual may wear appropriate 3-D 
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glasses to realize this effect or, in connection with emerging 3-D technologies, wear 
nothing additional at all. 

[0112] The partially real and partially simulated modular Interactive environment that 
has thus-far been described may be used in connection with a broad variety of 
applications, including simulations of military operations, architecture and interior 
design. It may be used in connection with theatrical presentations, location-based 
entertainment, theme parks, museums and in training and testing situations. Other 
applications are also embraced. The position of each display screen wall, such as the 
display screen walls 107 and/or 109, may be adjusted to precisely match the position 
that the walls need to be in so that the image they project is correctly located in the 
scene that the walls help to create. The image that is projected on each display screen 
wall, including its perspective, may additionally or instead be adjusted by the 
processing system to effectuate the same result. 

[0113] To help in this process, one or more additional sensors may be provided to 
sense the position of one or more of the display screen walls. Such sensors may or 
may not be attached to the display screen walls and may be optical, magnetic, utilize 
the GPS system, or may be of any other type. The information provided by these 
sensors may then be used to help position the walls precisely and/or to cause the 
image that is projected on them to be adjusted by the processing system, as 
discussed above. 

[0114] Although certain embodiments have now been described, it is to be 
understood that the features, components, steps, benefits and applications of these 
technologies are not so limited and that the scope of this application is intended to be 
limited solely to what is set forth in the following claims and to their equivalents. 
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